Skip to content

Fix test failures with OpenSSL 4.0.#1019

Open
junaruga wants to merge 6 commits intoruby:masterfrom
junaruga:wip/openssl-4.0-compatibility
Open

Fix test failures with OpenSSL 4.0.#1019
junaruga wants to merge 6 commits intoruby:masterfrom
junaruga:wip/openssl-4.0-compatibility

Conversation

@junaruga
Copy link
Member

This PR is based on the #1005, fixing the following test failures with OpenSSL master case. I don't intend to merge this PR. It's just show the result of the CI. I added the 5th commit on the #1005.

Use AES-256-CBC for OpenSSL to avoid errors on the error stack with OpenSSL 4.0. LibreSSL and AWS-LC don't support AES-256-CBC well, so use PBE-SHA1-3DES for them.

I haven't checked which commit of the openssl/openssl master branch caused the error.

This commit fixes the following test failures.
https://github.com/ruby/openssl/actions/runs/23093927984/job/67083113067?pr=1005

2) Failure: test_create_no_pass(OpenSSL::TestPKCS12)
/home/runner/work/openssl/openssl/test/openssl/utils.rb:140:in `teardown'
<[]> expected but was
<["error:0308010C:digital envelope routines::unsupported"]>

diff:
? ["error:0308010C:digital envelope routines::unsupported"]

However, I see other kind of test failure. I cannot reproduce this failure on my local, even when using the same commit of the OpenSSL openssl/openssl@5f084f8.

https://github.com/junaruga/ruby-openssl/actions/runs/23562223843/job/68604792678#step:11:590

1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'
     462: 
     463:   def test_private_encoding_encrypted
     464:     rsa = Fixtures.pkey("rsa2048")
  => 465:     encoded = rsa.private_to_der("aes-128-cbc", "abcdef")
     466:     asn1 = OpenSSL::ASN1.decode(encoded) # PKCS #8 EncryptedPrivateKeyInfo
     467:     assert_kind_of OpenSSL::ASN1::Sequence, asn1
     468:     assert_equal 2, asn1.value.size
Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'

rhenium added 4 commits March 16, 2026 22:11
OpenSSL's master branch is changing functions to return const pointers
where the returned objects are not meant to be modified by the caller.

Update ossl_*_new() to take const pointers accordingly. Unfortunately,
*_dup() in older versions of OpenSSL and in LibreSSL/AWS-LC take
non-const pointers, so const casts are required.
ossl_ec_new() was removed in commit 94aeab2 (pkey: simplify
ossl_pkey_new(), 2017-03-16), but it forgot to remove the declaration
while doing so.
ASN1_STRING has been made opaque in OpenSSL's master branch. Use the
new accessor functions instead of accessing fields directly.

Other uses of ASN1_STRING fields were already updated in
<ruby#978>. This patch converts the
remaining ones, which require the new functions added in OpenSSL 4.0
and were not available at that time.
OpenSSL master added support for RFC 7919 groups in TLS 1.2. They are
preferred over SSLContext#tmp_dh= or #tmp_dh_callback= values if the
client advertises them in the supported_groups extension.
@junaruga
Copy link
Member Author

Here is the error.

https://github.com/ruby/openssl/actions/runs/23562768731/job/68606686661?pr=1019#step:11:590

1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'
     462: 
     463:   def test_private_encoding_encrypted
     464:     rsa = Fixtures.pkey("rsa2048")
  => 465:     encoded = rsa.private_to_der("aes-128-cbc", "abcdef")
     466:     asn1 = OpenSSL::ASN1.decode(encoded) # PKCS #8 EncryptedPrivateKeyInfo
     467:     assert_kind_of OpenSSL::ASN1::Sequence, asn1
     468:     assert_equal 2, asn1.value.size
Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'

@junaruga
Copy link
Member Author

I got the following CI log when I ran without the 5th commit (my commit) now. Interestingly there is no error about Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error in the log. That means the 5th commit modifying test/openssl/test_pkcs12.rb caused the test_private_encoding_encrypted error in the test/openssl/test_pkey_rsa.rb. It is weird. Do I need to clean something up in the teardown method in the test/openssl/test_pkcs12.rb?

https://github.com/junaruga/ruby-openssl/actions/runs/23566877802/job/68620449742

@junaruga
Copy link
Member Author

There are differences between my local and GitHub Actions openssl-master environment. The differences may make the GitHub Actions specific test failure.

My local

OS: Fedora Linux 43

Ruby version

$ which ruby
~/.local/ruby-4.1.0dev-debug-dcfbbdc38c-openssl-4.0.0-dev-1cb0d36b39/bin/ruby

$ ruby -v
ruby 4.1.0dev (2026-01-08T09:48:38Z master dcfbbdc38c) +PRISM [x86_64-linux]

GitHub Actions openssl-master

OS: Ubuntu 24.04.4 LTS

Ruby version:

ruby 3.0.7p220 (2024-04-23 revision 724a071175) [x86_64-linux]

@junaruga
Copy link
Member Author

junaruga commented Mar 26, 2026

Oh I just noticed the test_private_encoding_encrypted error in test/openssl/test_pkey_rsa.rb rake test_fips step. So, that means that rake test passed by my 5th commit, then CI moved to the next step rake test_fips.

https://github.com/ruby/openssl/actions/runs/23562768731/job/68606686661?pr=1019#step:11:600

Use AES-256-CBC for OpenSSL to avoid errors on the error stack in teardown with
OpenSSL 4.0. Use AES-256-CBC for LibreSSL and AWS-LC, as they don't support
AES-256-CBC well.

This commit fixes the following test failures.
https://github.com/ruby/openssl/actions/runs/23093927984/job/67083113067?pr=1005

```
2) Failure: test_create_no_pass(OpenSSL::TestPKCS12)
/home/runner/work/openssl/openssl/test/openssl/utils.rb:140:in `teardown'
<[]> expected but was
<["error:0308010C:digital envelope routines::unsupported"]>

diff:
? ["error:0308010C:digital envelope routines::unsupported"]
```
@junaruga junaruga changed the title test_pkcs12.rb: Fix test failures with OpenSSL 4.0. Fix test failures with OpenSSL 4.0. Mar 26, 2026
@junaruga
Copy link
Member Author

For the est_private_encoding_encrypted error in test/openssl/test_pkey_rsa.rb .

$ cat test.rb
require 'openssl'

OpenSSL.debug = true
rsa = OpenSSL::PKey.read(File.read('test/openssl/fixtures/pkey/rsa2048.pem'))
rsa.private_to_der('aes-128-cbc', 'abcdef')
$ OPENSSL_DIR=$HOME/.local/openssl-4.1.0-dev-fips-debug-5f084f8a99
$ OPENSSL_CONF=$OPENSSL_DIR/ssl/openssl_fips.cnf \
  ruby -I ./lib test.rb
...
test.rb:5: warning: error on stack: error:1C8000FE:Provider routines:kdf_pbkdf2_set_ctx_params:password strength too weak
test.rb:5: warning: error on stack: error:11800067:PKCS12 routines:PKCS12_item_i2d_encrypt_ex:encrypt error
test.rb:5: warning: error on stack: error:11800067:PKCS12 routines:PKCS8_set0_pbe_ex:encrypt error
test.rb:5:in 'OpenSSL::PKey::PKey#private_to_der': i2d_PKCS8PrivateKey_bio: encrypt error (OpenSSL::PKey::PKeyError)
OpenSSL error queue reported 3 errors:
error:1C8000FE:Provider routines:kdf_pbkdf2_set_ctx_params:password strength too weak
error:11800067:PKCS12 routines:PKCS12_item_i2d_encrypt_ex:encrypt error
error:11800067:PKCS12 routines:PKCS8_set0_pbe_ex:encrypt error
	from test.rb:5:in '<main>'

It seems the test failed because the length of the password (abcdef): 6 is less than KDF_PBKDF2_MIN_PASSWORD_LEN (= 8).

$ gdb --args ruby -I ./lib test.rb
(gdb) set env OPENSSL_CONF /home/jaruga/.local/openssl-4.1.0-dev-fips-debug-5f084f8a99/ssl/openssl_fips.cnf
(gdb) b kdf_pbkdf2_set_ctx_params
...
(gdb) f
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10)
    at providers/implementations/kdfs/pbkdf2.c:369
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
(gdb) l
364	    }
365	
366	    if (p.pw != NULL) {
367	        if (ctx->lower_bound_checks != 0
368	            && p.pw->data_size < KDF_PBKDF2_MIN_PASSWORD_LEN) {
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
370	            return 0;
371	        }
372	        if (!pbkdf2_set_membuf(&ctx->pass, &ctx->pass_len, p.pw))
373	            return 0;
(gdb) f
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:369
369	            ERR_raise(ERR_LIB_PROV, PROV_R_PASSWORD_STRENGTH_TOO_WEAK);
(gdb) p KDF_PBKDF2_MIN_PASSWORD_LEN
$3 = 8
(gdb) p p.pw->data_size
$4 = 6

The stack trace is below.

(gdb) bt
#0  kdf_pbkdf2_set_ctx_params (vctx=0x8e0240, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:369
#1  0x00007fffcd22edb5 in kdf_pbkdf2_derive (vctx=0x8e0240, key=0x7fffffffaf90 "\320\300\215", keylen=16, params=0x7fffffffae10) at providers/implementations/kdfs/pbkdf2.c:312
#2  0x00007fffcd768066 in EVP_KDF_derive (ctx=0x8e0220, key=0x7fffffffaf90 "\320\300\215", keylen=16, params=0x7fffffffae10) at crypto/evp/kdf_lib.c:145
#3  0x00007fffcd7718f2 in ossl_pkcs5_pbkdf2_hmac_ex (pass=0x7fffffffb3d0 "abcdef", passlen=6, salt=0x8dc6b0 "4\020\005\232o\025\351\330\324_0\223\232Ir>", saltlen=16, iter=2048, digest=0x8ae4a0, keylen=16,
    out=0x7fffffffaf90 "\320\300\215", libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/evp/p5_crpt2.c:61
#4  0x00007fffcd772235 in PKCS5_v2_PBKDF2_keyivgen_ex (ctx=0x85ff90, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc090, c=0x0, md=0x0, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/evp/p5_crpt2.c:245
#5  0x00007fffcd771e06 in PKCS5_v2_PBE_keyivgen_ex (ctx=0x85ff90, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc070, c=0x0, md=0x0, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/evp/p5_crpt2.c:163
#6  0x00007fffcd7632ac in EVP_PBE_CipherInit_ex (pbe_obj=0x7fffcdca8268 <nid_objs+6440>, pass=0x7fffffffb3d0 "abcdef", passlen=6, param=0x8dc070, ctx=0x85ff90, en_de=1, libctx=0x7fffcdd16ee0 <default_context_int>,
    propq=0x0) at crypto/evp/evp_pbe.c:144
#7  0x00007fffcd89b29d in PKCS12_pbe_crypt_ex (algor=0x8da460, pass=0x7fffffffb3d0 "abcdef", passlen=6, in=0x8dc1e0 "0\202\004\276\002\001", inlen=1218, data=0x8dabf8, datalen=0x8dabf0, en_de=1,
    libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/pkcs12/p12_decr.c:39
#8  0x00007fffcd89ba22 in PKCS12_item_i2d_encrypt_ex (algor=0x8da460, it=0x7fffcdc9a780 <local_it>, pass=0x7fffffffb3d0 "abcdef", passlen=6, obj=0x8dafe0, zbuf=1, ctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/pkcs12/p12_decr.c:209
#9  0x00007fffcd89f997 in PKCS8_set0_pbe_ex (pass=0x7fffffffb3d0 "abcdef", passlen=6, p8inf=0x8dafe0, pbe=0x8da460, ctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0) at crypto/pkcs12/p12_p8e.c:76
#10 0x00007fffcd89f8da in PKCS8_encrypt_ex (pbe_nid=-1, cipher=0x8d0120, pass=0x7fffffffb3d0 "abcdef", passlen=6, salt=0x0, saltlen=0, iter=0, p8inf=0x8dafe0, libctx=0x7fffcdd16ee0 <default_context_int>, propq=0x0)
    at crypto/pkcs12/p12_p8e.c:51
#11 0x00007fffcd9a7fc5 in p8info_to_encp8 (p8info=0x8dafe0, ctx=0x8cef20) at providers/implementations/encode_decode/encode_key2any.c:133
#12 0x00007fffcd9a8067 in key_to_encp8 (key=0x8adc60, key_nid=6, params=0x0, params_type=5, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20) at providers/implementations/encode_decode/encode_key2any.c:150
#13 0x00007fffcd9a8224 in key_to_epki_der_priv_bio (out=0x8da610, key=0x8adc60, key_nid=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", p2s=0x7fffcd9a96c0 <prepare_rsa_params>, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20)
    at providers/implementations/encode_decode/encode_key2any.c:217
#14 0x00007fffcd9a8397 in key_to_pki_der_priv_bio (out=0x8da610, key=0x8adc60, key_nid=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", p2s=0x7fffcd9a96c0 <prepare_rsa_params>, k2d=0x7fffcd9a98ce <rsa_prv_k2d>, ctx=0x8cef20)
    at providers/implementations/encode_decode/encode_key2any.c:266
#15 0x00007fffcd9a9f47 in key2any_encode (ctx=0x8cef20, cout=0x8da280, key=0x8adc60, type=6, pemname=0x7fffcdbdb571 "RSA PRIVATE KEY", checker=0x7fffcd9a9920 <rsa_check_key_type>,
    writer=0x7fffcd9a832b <key_to_pki_der_priv_bio>, pwcb=0x7fffcd7ad43d <ossl_pw_passphrase_callback_enc>, pwcbarg=0x8b31f8, key2paramstring=0x7fffcd9a96c0 <prepare_rsa_params>, key2der=0x7fffcd9a98ce <rsa_prv_k2d>)
    at providers/implementations/encode_decode/encode_key2any.c:1255
#16 0x00007fffcd9ab9ca in rsa_to_PrivateKeyInfo_der_encode (ctx=0x8cef20, cout=0x8da280, key=0x8adc60, key_abstract=0x0, selection=135, cb=0x7fffcd7ad43d <ossl_pw_passphrase_callback_enc>, cbarg=0x8b31f8)
    at providers/implementations/encode_decode/encode_key2any.c:1546
#17 0x00007fffcd7348e9 in encoder_process (data=0x7fffffffbcf0) at crypto/encode_decode/encoder_lib.c:711
#18 0x00007fffcd733203 in OSSL_ENCODER_to_bio (ctx=0x8b31c0, out=0x6f28a0) at crypto/encode_decode/encoder_lib.c:91
#19 0x00007fffcd893a36 in do_pk8pkey (bp=0x6f28a0, x=0x8b9900, isder=1, nid=-1, enc=0x7fffcdb87060 <aes_128_cbc>, kstr=0x0, klen=0, cb=0x7fffce019e5b <ossl_pem_passwd_cb>, u=0x7fffce1177b0, propq=0x0)
    at crypto/pem/pem_pk8.c:124
#20 0x00007fffcd893815 in i2d_PKCS8PrivateKey_bio (bp=0x6f28a0, x=0x8b9900, enc=0x7fffcdb87060 <aes_128_cbc>, kstr=0x0, klen=0, cb=0x7fffce019e5b <ossl_pem_passwd_cb>, u=0x7fffce1177b0) at crypto/pem/pem_pk8.c:59
#21 0x00007fffce036437 in do_pkcs8_export (argc=2, argv=0x7fffe92ff050, self=140736650639400, to_der=1) at ../../../../ext/openssl/ossl_pkey.c:871
#22 0x00007fffce036515 in ossl_pkey_private_to_der (argc=2, argv=0x7fffe92ff050, self=140736650639400) at ../../../../ext/openssl/ossl_pkey.c:900
#23 0x00007ffff791db1d in ractor_safe_call_cfunc_m1 (recv=140736650639400, argc=2, argv=0x7fffe92ff050, func=0x7fffce0364eb <ossl_pkey_private_to_der>) at vm_insnhelper.c:3711
#24 0x00007ffff791e773 in vm_call_cfunc_with_frame_ (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, calling=0x7fffffffc700, argc=2, argv=0x7fffe92ff050, stack_bottom=0x7fffe92ff048) at vm_insnhelper.c:3902
#25 0x00007ffff791ea3b in vm_call_cfunc_with_frame (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:3948
#26 0x00007ffff791eb64 in vm_call_cfunc_other (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:3974
#27 0x00007ffff791efa0 in vm_call_cfunc (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:4056
#28 0x00007ffff7921c4d in vm_call_method_each_type (ec=0x40cbd0, cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:4888
#29 0x00007ffff7922716 in vm_call_method (ec=0x40cbd0, cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:5014
#30 0x00007ffff7922914 in vm_call_general (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, calling=0x7fffffffc700) at vm_insnhelper.c:5058
#31 0x00007ffff792531c in vm_sendish (ec=0x40cbd0, reg_cfp=0x7fffe93fefa0, cd=0x757ab0, block_handler=0, method_explorer=mexp_search_method) at vm_insnhelper.c:6124
#32 0x00007ffff792d5c3 in vm_exec_core (ec=0x40cbd0) at insns.def:904
#33 0x00007ffff7947575 in rb_vm_exec (ec=0x40cbd0) at vm.c:2798
#34 0x00007ffff7948456 in rb_iseq_eval_main (iseq=0x7fffce09ff58) at vm.c:3064
#35 0x00007ffff77023ff in rb_ec_exec_node (ec=0x40cbd0, n=0x7fffce09ff58) at eval.c:283
#36 0x00007ffff7702567 in ruby_run_node (n=0x7fffce09ff58) at eval.c:321
#37 0x0000000000400519 in rb_main (argc=4, argv=0x7fffffffd908) at ./main.c:42
#38 0x0000000000400576 in main (argc=4, argv=0x7fffffffd908) at ./main.c:62

…FIPS.

OpenSSL 4.0.0 added a check for Password-Based Key Derivation Function 2
(PBKDF2) to require the minimal password length 8 in FIPS by the following
commit.
openssl/openssl@71ed0fc

This commit fixes the following test failure in OpenSSL 4.0 FIPS by changing
testing password length from 6 to 8..

```
1) Error: test_private_encoding_encrypted(OpenSSL::TestPKeyRSA): OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `private_to_der'
/home/runner/work/openssl/openssl/test/openssl/test_pkey_rsa.rb:465:in `test_private_encoding_encrypted'
     462:
     463:   def test_private_encoding_encrypted
     464:     rsa = Fixtures.pkey("rsa2048")
  => 465:     encoded = rsa.private_to_der("aes-128-cbc", "abcdef")
     466:     asn1 = OpenSSL::ASN1.decode(encoded) # PKCS ruby#8 EncryptedPrivateKeyInfo
     467:     assert_kind_of OpenSSL::ASN1::Sequence, asn1
     468:     assert_equal 2, asn1.value.size
Error: OpenSSL::PKey::PKeyError: i2d_PKCS8PrivateKey_bio: encrypt error
```
@junaruga junaruga force-pushed the wip/openssl-4.0-compatibility branch from 528c991 to f909fd5 Compare March 26, 2026 18:38
@junaruga junaruga marked this pull request as ready for review March 26, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants